{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Otto商品分类——Logistic 回归\n",
    "原始特征+tfidf特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，分别调用\n",
    "缺省参数LogisticRegression、\n",
    "LogisticRegression + GridSearchCV （可用LogisticRegressionCV代替）进行参数调优。\n",
    "\n",
    "Otto数据集是著名电商Otto提供的一个多类商品分类问题，类别数=9. 每个样本有93维数值型特征（整数，表示某种事件发生的次数，已经进行过脱敏处理）。 竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data\n",
    "\n",
    "第一名：https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/14335\n",
    "第二名：http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# 读取数据\n",
    "# 输入特征为：原始特征和tf_idf特征上\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "\n",
    "train1 = pd.read_csv(dpath +\"Otto_FE_train_org.csv\")\n",
    "train2 = pd.read_csv(dpath +\"Otto_FE_train_tfidf.csv\")\n",
    "\n",
    "#去掉多余的id\n",
    "train2 = train2.drop([\"id\",\"target\"], axis=1)\n",
    "train =  pd.concat([train1, train2], axis = 1, ignore_index=False)\n",
    "train.head()\n",
    "\n",
    "del train1\n",
    "del train2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = train['target']   \n",
    "X_train = train.drop([\"id\", \"target\"], axis=1)\n",
    "\n",
    "#保存特征名字以备后用（可视化）\n",
    "feat_names = X_train.columns \n",
    "\n",
    "#sklearn的学习器大多之一稀疏数据输入，模型训练会快很多\n",
    "#查看一个学习器是否支持稀疏数据，可以看fit函数是否支持: X: {array-like, sparse matrix}.\n",
    "#可自行用timeit比较稠密数据和稀疏数据的训练时间\n",
    "from scipy.sparse import csr_matrix\n",
    "X_train = csr_matrix(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 默认参数的Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [ 0.63005681  0.63058577  0.62210687]\n",
      "cv logloss is: 0.627583152651\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "#数据集比较大，采用3折交叉验证\n",
    "from sklearn.model_selection import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=3, scoring='neg_log_loss')\n",
    "#%timeit loss_sparse = cross_val_score(lr, X_train_sparse, y_train, cv=3, scoring='neg_log_loss')\n",
    "print ('logloss of each fold is: ',-loss)\n",
    "print ('cv logloss is:', -loss.mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原始特征：0.797465616286\n",
    "log特征：0.684053401541\n",
    "tfidf特征：0.646590780501\n",
    "\n",
    "原始特征 + tfidf特征：0.627583152651"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "目标函数为：J =  C* sum(logloss(f(xi), yi)) + penalty \n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同：\n",
    "设置参数搜索范围\n",
    "生成GridSearchCV的实例（参数）\n",
    "调用GridSearchCV的fit方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "\n",
    "#训练数据多，C可以大一点（更多相信数据）\n",
    "Cs = [ 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=3, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.606741859533\n",
      "{'penalty': 'l1', 'C': 10}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/Users/qing/anaconda2/lib/python2.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAELCAYAAAAoUKpTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VdXV+PHvujcThDCHeYYgJCgg\nERkEQZmcUOvI2ypa69BK6/Bqf1opKGirte9ba6V1qlMH0Zc6IA7MiKIgYRIIU0CQMBOGMGe46/fH\nOcEQMpwbcnJDsj7Pc57cnLv3vSsHclf2PuesLaqKMcYYU5pApAMwxhhT9VmyMMYYUyZLFsYYY8pk\nycIYY0yZLFkYY4wpkyULY4wxZfI1WYjICBFZJyIZIvJIMc//SUSWu9t6ETlQ6LnRIrLB3Ub7Gacx\nxpjSiV/3WYhIEFgPDAUygcXAKFVNL6H9L4GeqvpTEWkIpAGpgAJLgF6qut+XYI0xxpTKz5FFbyBD\nVTepag4wGbi6lPajgLfdx8OBmaq6z00QM4ERPsZqjDGmFH4mi5bA1kLfZ7r7TiMibYH2wJxw+xpj\njPFflI+vLcXsK2nO62Zgiqrmh9NXRO4C7gKIj4/v1aVLl/LEaYwxNdaSJUv2qmpiWe38TBaZQOtC\n37cCtpfQ9mbg3iJ9BxXpO69oJ1V9GXgZIDU1VdPS0sofrTHG1EAissVLOz+noRYDSSLSXkRicBLC\n1KKNROQcoAHwdaHd04FhItJARBoAw9x9xhhjIsC3kYWq5onIGJwP+SDwmqquFpEJQJqqFiSOUcBk\nLXRZlqruE5GJOAkHYIKq7vMrVmOMMaXz7dLZymbTUMYYEz4RWaKqqWW18/OchTHGRExubi6ZmZkc\nP3480qFUCXFxcbRq1Yro6Ohy9bdkYYypljIzM0lISKBdu3aIFHeBZc2hqmRlZZGZmUn79u3L9RpW\nG8oYUy0dP36cRo0a1fhEASAiNGrU6IxGWZYsjDHVliWKH5zpsbBkYYwxrpte+pqbXvq67IY1kCUL\ngNevcDZjjKlAderUOfl4xIgR1K9fnyuvvLLYtvfeey89evQgOTmZWrVq0aNHD3r06MGUKVPCes+l\nS5fy2WefnVHcxbET3MYYUwkefvhhjh49yksvvVTs85MmTQJg8+bNXHnllSxfvrxc77N06VJWrVrF\niBEVW3vVRhbGGFMJLr30UhISEsrVd8OGDQwfPpxevXoxcOBA1q9fD8DkyZPp1q0b3bt3Z/DgwRw7\ndowJEybwr3/9q1yjktLYyMIYU+098dFq0rdnl9kufYfTxst5i+QWdRl/VcoZx+bFXXfdxauvvkrH\njh1ZsGABY8aMYcaMGTzxxBPMmzePpk2bcuDAAWrVqsW4ceNYtWoVzz33XIXGYMnCGGOqsAMHDrBw\n4UKuu+66k/vy8vIA6N+/P7feeis33HADP/rRj3yNw5IFwIlDEF070lEYY3zidQRQMKJ45+6+foYT\nFlWlcePGxZ7DeOWVV1i0aBHTpk2je/fufPvtt77FYecs9mbAzhWQvS3SkRhjzGkaNGhA8+bNef/9\n9wEIhUKsWLECgE2bNtGnTx8mTpxIgwYN2LZtGwkJCRw6dKjC47Bk0bgT1G4E2ZlwaGekozHGVFMD\nBgzghhtuYPbs2bRq1Yrp072vujB58mRefPFFunfvTkpKCtOmTQPggQce4Nxzz+Xcc89lyJAhdOvW\njUsuuYQVK1bQs2dPO8Fd4eq3g6P7YO5TMPIvkY7GGFNNHD58+OTjL774wlOfdu3asWrVqlP2dejQ\nodjkMnXqaUsEkZiYiB8VuC1ZAETXgoTmsOyfcOE90LRyrnAwxlQtVelcRVVj01AF6rWG2Low47eR\njsQYY6ocSxYFgtFw8a9h42zImBXpaIwxpkqxZFHYBXdCg/bO6CKUH+lojDGmyrBkAXD7x84WFQND\nn4Dd6bDsH5GOyhhjqgxLFkV1HQmt+8Ccp5yb9YwxNYdVoC6RJYuiRGD4U3BkNyz4c6SjMcacxQpK\nlC9fvpy+ffuSkpLCeeedxzvvvHNaWytRfjZqlQrdroOvXoBet0O9lpGOyBhzFqtduzZvvfUWSUlJ\nbN++nV69ejF8+HDq169/so2VKD9bXToeNARznox0JMaYs1znzp1JSkoCoEWLFjRp0oQ9e/Z47m8l\nyquyBm2hzz2w4Hnna/PukY7IGFNenz4CO1eW3W6nW4jPy3mLZufCZU+HHco333xDTk4OHTt29NzH\nSpRXdQP+27mre/pjMPoj53yGMcaU044dO7jlllt48803CQS8TexYifKzQVw9GPQofPIQrP8Mzrks\n0hEZY8rD6wigYERx+8cVHkJ2djZXXHEFTz75JH369PHcz0qUny163QaNkpwb9fJzIx2NMeYslJOT\nw7XXXntyFBAOK1F+tghGw9AJkLUBlrwR6WiMMWehd999l/nz5/PGG2+cvCQ2nKudqkKJclHVCnux\nSEpNTVU/yvICoApvXuXc2f2rZc70lDGmSluzZg1du3YNr5OP01BVQXHHRESWqGpqWX1tZOGFCAx7\n0lnz4ov/jXQ0xhi/FJT+MaexZIGz7m7B2rslatEDut8MC/8G+7dUTmDGGFNFWLIIxyW/BQnA7AmR\njsQY40F1mWavCGd6LCxZhKNeS+g3BlZNgcwlkY7GGFOKuLg4srKyLGHgJIqsrCzi4uLK/Rq+3mch\nIiOAPwNB4FVVPe1iZxG5EXgcUGCFqv6Xu/8PwBU4CW0mcJ9WhX/1/vfBkjdhxmNw+6d2o54xVVSr\nVq3IzMwMq6xGdRYXF0erVq3K3d+3ZCEiQWASMBTIBBaLyFRVTS/UJgl4FOivqvtFpIm7vx/QHzjP\nbfolcDEwz694PYtNgMG/gWn3w5qPIHlkpCMyxhQjOjqa9u3bRzqMasPPaajeQIaqblLVHGAycHWR\nNncCk1R1P4Cq7nb3KxAHxACxQDSwy8dYw9PzFkjsCrPGQ15OpKMxxhjf+ZksWgJbC32f6e4rrDPQ\nWUQWiMhCd9oKVf0amAvscLfpqrrGr0DDnt0KRjmX0u7bBItf9ScoY4ypQvxMFsVN5hf9VI4CkoBB\nwCjgVRGpLyKdgK5AK5wEc4mIDDztDUTuEpE0EUkr77zk1n1HWbktmwNHwxwhJA2BjpfA5884918Y\nY0w15meyyARaF/q+FbC9mDYfqmquqn4HrMNJHtcCC1X1sKoeBj4FTqu8paovq2qqqqYmJiaWK8im\ndeNQlC37jpKTFwqv87An4UQ2zP9jud7bGGPOFn4mi8VAkoi0F5EY4GZgapE2HwCDAUSkMc601Cbg\ne+BiEYkSkWick9u+TEPFRAVo27A2x3NDvPHVd+F1bpoCPX4M37wMWRv9CM8YY6oE35KFquYBY4Dp\nOB/076rqahGZICIFlxBNB7JEJB3nHMXDqpoFTAE2AiuBFTiX1H7kV6z1a8dQv1Y0z8/OYPeh4+F1\nvmQsBGNg9hP+BGeMMVWAFRLEKfdxPDef9B3ZXN2jJX+8IcxV8eY9A/N+Bz+dDm2816k3xphIs0KC\nYYqLDvLTi9ozZUkmy77fH17nfmMgobmzol41Sb7GGFOYJYtCfnlJEokJsTz+UTqhUBgf+jHxTt2o\nbWmw6j/+BWiMMRFiyaKQOrFRPDKiCyu2HuC9ZdvC69z9ZmcB91lPQG6Y5z2MMaaKs2RRxLU9W9Kz\nTX2e/nQth46HsYxqIOhcSnvwe1j0on8BGmNMBFiyAN65uy/v3N0XgEBAePyqFPYePsELczLCe6EO\ngyBpOHzxP3Akq8LjNMaYSLFkUYzuretzQ69WvLbgOzbtORxe52ETIecIfH5agV1jjDlrWbIowa9H\ndCEuKsjEaellNy4s8RzodRukvQZ7N/gSmzHGVDZLFiVITIjlV5cmMXfdHuasDbPg7aBHIaoWzBzn\nT3DGGFPJLFmUYnS/dnRIjGfitDXh1Y2qkwgDHoR1n8B3X/gXoDHGVBJLFqWIiQow7spkvtt7hNcX\nhFk3qs/PoV5rZ0W9UJgFCo0xpoqxZFGGQec04dIuTXh+9gZ2Z4dx/0R0Lbh0HOxYASvf9S9AY4yp\nBJYsPPjtlcnk5ivPfLYuvI7drocWPWH2BMg56k9wxhhTCSxZeNCucTw/vag9/1kaZt2oQACG/w6y\nt8HCSf4FaIwxPrNk4dGYSzrRJCGWx6euDq9uVNt+0OVK+PI5OFR1lhE3xphwWLLwqE5sFI9c1oUV\nmQf5z9LM8DoPnQB5x50y5sYYcxayZBGGa3o4daOe+WxdeHWjGnWEC+6EpW/BrjBv8jPGmCrAkkUY\nCupGZR05wV/CrRt18a8hNsFu1DPGnJUsWYSpoG7U6wu+Y2M4daNqN4SBD0PGTNg4x78AjTHGB5Ys\nyuHh4eWsG9X7LmjQDqaPhVC+L7EZY4wfLFmUQ2JCLPcNSWJeuHWjomJhyOOwezUs/5df4RljTIWz\nZFFOt/Z16kZN+CidE3lhjBKSr4HWF8KcJ+FEmOXPjTEmQixZlFNB3ajNWUd5fcFm7x1FYNhTcHgX\nfPW8b/EZY0xFsmRxBgad04QhXZvwl3DrRrW+AFKuhQXPQ/Z2/wI0xpgKYsniDI29wqkb9fRna8Pr\nOORx0HyY85QfYRljTIWyZHGG2jWO544B7Xlv6bbw6kY1aAcX3u2c6N7xrW/xGWNMRbBkUQHuHVzO\nulEDHoJaDWDGWNAw+hljTCWzZFEBCteNmhJO3aha9WHQI/Dd57Bhhn8BGmPMGbJkUUGu6dGS89vU\n5w+frSM7nLpRqT+Fhh1hxm8hP8+/AI0x5gxYsqgggYDw+Ei3btTsDd47BqOdqrR718HSN/0L0Bhj\nzoAliwp0Xqv63NirNa8v2Bxe3aguV0Db/jD3d3A8278AjTGmnCxZVLCHR5xDreggEz5KR72etBaB\nYU/C0b3w5Z/8DdAYY8rBkkUFa1zHqRv1+fo9zFm723vHlufDeTfBwr/Cga3+BWiMMeVgycIHt/Zt\nR8fEeCZOC7Nu1KXuWhezJ/gTmDHGlJMlCx/ERAUYd1UKm7OO8tqXm713rNcK+t4LK9+FbUt8i88Y\nY8Lla7IQkREisk5EMkTkkRLa3Cgi6SKyWkT+XWh/GxGZISJr3Ofb+RlrRbu4cyJDujbhhTlh1o3q\nfz/EJzprXtiNesaYKsK3ZCEiQWAScBmQDIwSkeQibZKAR4H+qpoC3F/o6beAZ1W1K9AbCOMEQNVQ\nrrpRcXVh0KPw/Vew9mP/gjPGmDD4ObLoDWSo6iZVzQEmA1cXaXMnMElV9wOo6m4AN6lEqepMd/9h\nVT3qY6y+KFw3amk4daPOHw2JXZz1uvNy/AvQGGM88jNZtAQKX9aT6e4rrDPQWUQWiMhCERlRaP8B\nEXlPRJaJyLPuSOUUInKXiKSJSNqePXt8+SHO1JjBnWhaN8y6UcEoGDoR9m2EtNf8DdAYYzzwM1lI\nMfuKflpGAUnAIGAU8KqI1Hf3DwAeAi4AOgC3nfZiqi+raqqqpiYmJlZc5BUo3q0b9W3mQaYsCaNu\nVNJQ6DAIPn8ajoUxKjHGGB/4mSwygdaFvm8FFF3pJxP4UFVzVfU7YB1O8sgElrlTWHnAB8D5Psbq\nq5N1o6av9V43quBGvWMHYP4f/Q3QGGPK4GeyWAwkiUh7EYkBbgamFmnzATAYQEQa40w/bXL7NhCR\nguHCJUC6j7H6SqSgblROeHWjmp0LPX8M37wM+77zL0BjjCmDb8nCHRGMAaYDa4B3VXW1iEwQkZFu\ns+lAloikA3OBh1U1S1XzcaagZovISpwprVf8irUyFK4blbE7jLpRg8dCIApmP+FfcMYYUwbxXL+o\niktNTdW0tLRIh1GqvYdPMPjZefRs24A3b78AkeJO6xRj7u+dcxd3zITWvf0N0hhTo4jIElVNLaud\n3cFdiQrqRs1fv4fZa8K4baT/r6BOM5j+G7tRzxgTEZYsKtnofm7dqI/DqBsVEw+XjIXMxbD6fX8D\nNMaYYliyqGTRwQDjr0phS7h1o3r8FzTtBrMeh7wTfoVnjDHFsmQRAQM7JzKka1P+MmcDu7zWjQoE\nnUtpD2yBRS/5G6AxxhQRdrIQkYCI1PUjmJrkt1d2JS9feebTMOpGdRwMnYY6910c3edfcMYYU4Sn\nZCEi/xaRuiISj3O/wzoRedjf0Kq3to3i+dmA9ry3bBtLtoRxh/awiZBzCD5/xr/gjDGmCK8ji2RV\nzQauAT4B2gC3+BZVDXGvWzfqiY/CqBvVpKtTaHDxq7A3w98AjTHG5TVZRItINE6y+FBVczm9zpMJ\nU3xsFI9e1jX8ulGDfwNRcTBrvH/BGWNMIV6TxUvAZiAemC8ibYFsv4KqSa7u0YJebRuEVzeqThO4\n6AFYOw02f+lvgMYYg8dkoarPq2pLVb1cHVtwazqZMyMiPH6VUzfq+Vlh1I3qey/UbQnTH4NQyL8A\njTEG7ye473NPcIuI/F1EluIU9zMV4NxW9bgptTVvfBVG3ajoWnDpONixHFZN8TdAY0yN53Ua6qfu\nCe5hQCJwO/C0b1HVQA8NP4daMUEmTEvHc72uc2+E5j1g1hOQe8zfAI0xNZrXZFFQ8e5y4HVVXUHx\nixuZcmpcJ5b7h3QOr25UIADDn4LsTFj4V38DNMbUaF6TxRIRmYGTLKaLSAJgE+UV7Na+benUpE54\ndaPaXQTnXAFf/AkOV82lZY0xZz+vyeIO4BHgAlU9CsTgTEWZChQdDDDuymS2ZB3l71+GsdjR0AmQ\ndwzm/c6/4IwxNZrXq6FCOMuijhWRPwL9VPVbXyOroQZ2TmRoclNemJPhvW5U406QegcseQN2h1E+\nxBhjPPJ6NdTTwH04pT7SgV+JyO/9DKwmG3uFUzfq6XDqRl38/yAmAWaO8y8wY0yN5XUa6nJgqKq+\npqqvASOAK/wLq2Zr2yieOwe25/1l21iyxWPBwPhGMPC/YcN02DTP1/iMMTVPOFVn6xd6XK+iAzGn\n+sUgp27U41PTvdeN6n031G8D08dCyOMJcmOM8cBrsvg9sExE3hCRN4ElgJ1N9VFB3aiV2w7yf0u2\neusUHQdDHoddK2HF236GZ4ypYbye4H4b6AO85259VXWyn4GZQnWjPlvHwWMe60al/AhaXQCzJ0LO\nEX8DNMbUGKUmCxE5v2ADmgOZwFaghbvP+EhEeGJkCvuO5vD8bI91o0Rg2FNweCd89Rd/AzTG1BhR\nZTz/P6U8p1h9KN91a+nUjXrzq82M6t2aTk0Syu7U5kJIvgYW/Bl63QYJzXyP0xhTvZU6slDVwaVs\nligqSUHdqCc+CqNu1JDxkJ8Lc570NzhjTI3g9T6LHxWzXSoiTfwO0PxQN+qLDXuZ5bVuVMMOcOHd\nsOyfsHOVvwEaY6q9cMp9vAr82N1eAR4EFoiILa9aCU7WjZqWzvFcj5fFDnwIatWHGWPB64jEGGOK\n4TVZhICuqnqdql4HJAMngAuB/+dXcOYH0cEA469K5vt9YdSNqtXAubN701zImOVvgMaYas1rsmin\nqrsKfb8b6Kyq+wCP13SaMzUgyakbNWluBjsPeqwblXqHMyU1Yyzk5/kboDGm2vKaLL4QkWkiMlpE\nRgNTcdbijgcO+BeeKeq3VySTF1Ke/nSNtw5RMU5V2j1rYdlb/gZnjKm2vCaLe4HXgR5AT+BN4F5V\nPaKqthZ3JWrTqDZ3DmjPB8u3e68b1eVKaNMP5v4OThzyN0BjTLXk9Q5uBb4E5gCzgPnq+RpOU9F+\nMagTzerG8fjUdPK91I0SgeFPwpE98OVz/gdojKl2vF46eyPwDXA9cCOwSESu9zMwU7L42CgevbyL\nUzcqzWPdqJa94Nwb4OsX4GCmvwEaY6odr9NQj+GskjdaVW8FegO/9S8sU5aR3VuQ2rYBz04Po27U\npeOcS2hnT/Q3OGNMteM1WQRUtfDdYFle+orICBFZJyIZIvJICW1uFJF0EVktIv8u8lxdEdkmIi94\njLPGEBEed+tG/XmWx7pR9dtA31/At5Nh+zJ/AzTGVCtek8VnIjJdRG4TkduAj4FPSusgIkFgEnAZ\nzn0Zo0QkuUibJOBRoL+qpgD3F3mZicDnHmOscbq1rMfNF7Tmra83k7Hb44nrix6E2o2dNS/stJMx\nxiOvJ7gfBl4GzgO6Ay+ralk34/UGMlR1k6rmAJOBq4u0uROYpKr73fc5OXoRkV5AU2CGlxhrqoeG\nhVk3Kq4uDHoEtnwJ6z71P0BjTLXgeaU8Vf2Pqj6oqg+o6vseurTEKWdeINPdV1hnoLOILBCRhSIy\nAkBEAjgVbx/2Gl9N1ahOLA+4daNmpu8quwNAr9uhcWeY+Vun2KAxxpShrPUsDolIdjHbIRHJLuO1\npZh9Rf/0jQKSgEHAKOBVEakP/AL4RFVLvdRHRO4SkTQRSduzZ08Z4VRft/RtS1KTOjz58RpvdaOC\nUTB0ImRlQNrr/gdojDnrlVWiPEFV6xazJahq3TJeOxNoXej7VsD2Ytp8qKq5qvodsA4nefQFxojI\nZuCPwK0i8nQx8b2sqqmqmpqYmFhGONVXdDDAuHDrRnUeDu0HwrzfwzG7Cd8YUzrP01DlsBhIEpH2\nIhID3IxTJqSwD4DBACLSGGdaapOq/lhV26hqO+Ah4C1VLfZqKuMYkJTIsHDqRhWsqHdsP3xR2hpX\nxhjjY7JQ1TxgDDAdWAO8q6qrRWSCiIx0m00HskQkHZgLPKyqWX7FVN2NDbduVPPzoMd/waIXYf9m\nX2MzxpzdpLpU7UhNTdW0tLRIhxFxf5y+jhfmZjDlnr6ktmtYdofs7fD8+dDlcrj+Nf8DNMZUKSKy\nRFVTy2rn5zSUiYBfDO7o1I36aLW3ulF1W0C/X8Kq/0CmJVtjTPEsWVQztWOculGrtmV7rxvV/z6o\n0xSm/8Zu1DPGFMuSRTU0snsLLmjXgD94rRsVWwcGPwZbF0H6h/4HaIw561iyqIZEhPFXpbA/nLpR\nPX8CTVJg1njIO+FvgMaYs44li2rKqRvVhje/3syGXR7qRgWCMGyic1XUN6/4HZ4x5ixjyaIae2hY\nZ2rHBJkwzWPdqE6XQqchMP8PcNTjKnzGmBrBkkU11qhOLA8ODbNu1NCJztKr85/1NzhjzFnFkkU1\n95M+Tt2oiR+ne6sb1TQZet7iTEVlbfQ/QGPMWcGSRTUXHQww/qoUtu475r1u1ODHICrWOdltjDFY\nsqgRLkpqzPCUprwwJ4MdB4+V3SGhKfS/H9Z8BFu+8j9AY0yVZ8mihhh7RTL5qjz96VpvHfreCwkt\nYPpjEAr5G5wxpsqzZFFDtG5Ym7sGdODD5dtJ2+zhSqeY2nDpONi+FFa/53+AxpgqzZJFDVJQN2r8\nVI91o867CZqdB7OegFwPZc+NMdWWJYsapKBu1Ort2bzrpW5UIADDn4KD38Oiv/2w//UrnM0YU2NY\nsqhhCupGPTt9HQePeqgb1X4gdL4MvvhfOLLX/wCNMVWSJYsapnDdqOdmr/fWaegEyDniLMFqjKmR\nLFnUQAV1o976eou3ulGJnSH1p5D2OuxZ53+Axpgqx5JFDfXQsM7ExwR54iOPdaMGPQIx8TDTbtQz\npiayZFFDNaoTywNDO/Nlxl5meKkbFd8YBjwI6z+F4wf8D9AYU6VYsqjBftKnLZ2b1uFJr3WjLvw5\n1GsD+7+zFfWMqWEsWdRghetGvfrFJg8d4mDIeOdk976NcMDjsq3GmLOeJYsarn8np27UpLkbvdWN\n6nYd1GkGh3fCn7vDlJ/CtqX+B2qMiShLFuZk3ajff+KhbpQINOoELVOhz89hw0x4ZTC8fjms/cTq\nSBlTTVmyMLRuWJu7B3Zg6ortLPZSNwogKs65u/uB1TDsKTjwPUweBZMugMV/h5yj/gZtjKlUliwM\nAD8f1JHm9eIY/6HHulEF4upCvzHwq+Vw/WsQWxc+fhD+lAJznoLDu/0L2hhTaSxZGKCgblRX0ndk\n887icpy4DkY55zPunAO3fwpt+jpLs/6pG3w4BnavqfigjTGVxpKFOemq85rTu11D/jjDY92o4ohA\n234w6t8wJg16/gRWToG/9oF/Xgcb59plt8achSxZmJNEhPEjkzlwNIc/zfJYN6o0jTvBlf/rnNcY\nPBZ2fAv/uAZeHADL34a8nDN/D2NMpbBkYU6R0qIeN/duwz8WbmG9l7pRXsQ3gosfhvtXwsgXIJQH\nH9wDfz7PqWZ7bH/FvI8xxjfiqS7QWSA1NVXT0tIiHUa1sO9IDoOencu5rerxzzsuREQq9g1UIWM2\nfP0X2DQPouOd6ao+P4eG7Sv2vYwxpRKRJaqaWlY7G1mY0zSMj+HBoZ1ZkJHF9NUe6kaFSwSShsCt\nH8I9X0LySEh7Df5yPrxzC2z9puLf0xhzRixZmGKFXTeqvJqdC9e+6ExR9b8Pvvsc/j4UXh0K6R9C\nyMf3NsZ4ZsnCFCvKrRuVuf8Yr8z3UDfqTNVtDkMehwfS4bI/wJHd8O6tzmhj0Utw4rD/MRhjSuRr\nshCRESKyTkQyROSREtrcKCLpIrJaRP7t7ushIl+7+74VkZv8jNMUr3+nxoxIacZf551aN+qml77m\nppe+9udNY+vAhXfDL5fCjW9BfBP49Nfwp2RnLY3s7f68rzGmVL4lCxEJApOAy4BkYJSIJBdpkwQ8\nCvRX1RTgfvepo8Ct7r4RwHMiUt+vWE3JHruiq/e6URUpEITkq+FnM+GOmdBhEHz1PDx3Hrx3N+xc\nWbnxGFPD+Tmy6A1kqOomVc0BJgNXF2lzJzBJVfcDqOpu9+t6Vd3gPt4O7AYSfYzVlKB1w9rc49aN\n+uY7j3WjKjyI3s4o45dL4YI7YM1H8OJF8OZIp5BhNbmiz5iqzM9k0RIoXDci091XWGegs4gsEJGF\nIjKi6IuISG8gBtjoW6SmVPe4daMenxpm3aiK1rA9XPYMPLjaOb+xdz3863rn7vClb0Hu8cjFZkw1\n52eyKO7i/KKfNFFAEjAIGAW8Wni6SUSaA/8AblfV02pfi8hdIpImIml79uypsMDNqQrXjZq8+PtI\nhwO1GsBFD8B938K1L0EgGqb+Ep7rBp//AY5kRTpCY6odP5NFJtC60PetgKJnJzOBD1U1V1W/A9bh\nJA9EpC7wMTBWVRcW9waq+rLatnMJAAASxUlEQVSqpqpqamKizVL56WTdqOnryMuvImtWRMVA95vh\nni+cezaa94C5TzkVb6c9AHszIh2hMdWGn8liMZAkIu1FJAa4GZhapM0HwGAAEWmMMy21yW3/PvCW\nqv6fjzEajwrqRh08lkvmAQ8r6lUmEecE+E+mwC8WwbnXw7J/wgup8PYo2LzAzmsYc4Z8SxaqmgeM\nAaYDa4B3VXW1iEwQkZFus+lAloikA3OBh1U1C7gRGAjcJiLL3a2HX7Eab1Ja1GNU7zbsyj7BgaM5\nkT1/UZImXeDqF5zihQMfhu8XwhuXO6v5rZwC+eWspmtMDWe1oUxY9h3J4cLfzSI3X2kYH8MlXZow\nLLkpA5ISqRUTjHR4p8s5CivehoV/hawMqNfauY/j/NHOwk3G1HBea0NZsjBhu/5vX3HgWC7dWtRl\n9trdHDqeR1x0gAFJiQxLbsqlXZvSMD4m0mGeKhSC9Z/B1y/AlgUQkwC9RsOF90D91mX3N6aa8pos\noiojGFO9BANCo/gYnru5J7n5Ib75bh8zVu9kRvouZqbvIiCQ2q4hw5KbMiy5GW0a1Y50yBAIQJfL\nnW3bUidpLPybs6VcA33HQMvzIx2lMVWWjSxM2ApKfbxzd99T9qsqq7dnn0wca3c662F0aZbAUDdx\ndGtZt+JLnpfXga2w6EVY8ibkHIK2/Z2k0XmEk1yMqQFsGsr4pqRkUdTWfUeZkb6LGat3snjzPkIK\nzevFMTS5KUOTm3Jh+0bERFWBD+Xj2c5NfYtehINboVEn6PML6D4KYqrAqMgYH1myML7xmiwK23ck\nhzlrdzMzfSefr9/D8dwQCXFRDD6nCcNSmnJx50QS4qL9Ctmb/DxI/8CZotq+DGo1hAt+Br3vhDpN\nIhubMT6xZGF8U55kUdixnHy+zNjLzPSdzFqzm31HcogJBujbsdHJUUfTunEVGXJ4VGHLV07SWPcp\nBGPgvBudKaomXcJ/vdevcL7e/nHFxmlMBbAT3KbKqhUTPJkU8kPK0u/3nzzPMfaDVYz9YBU9Wtdn\naHJThqc0pWNinco9zyEC7fo7294MWDgJlv8blv0DOg2FfmOg/cVOO2NqCBtZmLCd6ciiJKrKht2H\nmeme51iReRCA9o3jGeYml55tGhAMROBD+kgWpP0dvnkZjuyBpudC33uh23VO2ZHS2MjCVGE2DWXO\nejsPHmfmGidxLNyURW6+0rhODEO6Oomjf6fGxEVX8o2Aucdh5bvw9STYsxYSmkPvuyD1dqfAYXEs\nWRgfnekfbzYNZc56zerFcUufttzSpy3Zx3OZt24PM1bvZNq3O5i8eCu1Y4IMTEpkWEpTLunShPq1\nK+FGwOg4OP9W6HkLZMyCr/4Cs5+A+X+Enj+BPj93SqkbU81YsjBnhbpx0Yzs3oKR3VtwIi+fhZv2\nMTN9JzPTd/HZ6p0EA0Lvdg1Pngtp3dDnS15FIGmos+341hlppP0dFr8CXa6Efr90Fm0yppqwaShz\nVguFlJXbDjLDTRzrdx0GoGvzuifPc6S0qKQbAbO3w6KXYMnrcPwgtOrtnAxf+JKTXGwayhO/zolV\nV5U1DWXJwlQrm/cecU6Qp+8kbct+VKFl/VruHeRNuaB9Q6KDPt8IeOKwUyJ94V/hwBaIinPOZ/S+\nC+q2hLotoF5LSGjhTGuZU1iyCM/q310EQMpvvixXf0sWpsbbe/gEc9bsZkb6Tr7YsJcTeSHq1Yo+\nWSl3YOdE4mN9nIkN5cPaafDhGMg9CqG809vUbuQkj4IkcvJxwdYcYuL9i7EKOtMPv5qmspKFnbMw\n1VbjOrHceEFrbrygNUdz8pi/fi8z03cxe+0u3l+2jZioABd1anyyUm5iQmzFBhAIQvLVrP7gWYiC\nlIemQ/YOyN5WaNvubtsgczEcLWZJ2Lj6xSQTd3RS8Dg2oWJjN6YISxamRqgdE8WIbs0Y0a0Zefkh\n0rbsZ8ZqZ7pqztrdiKzk/DYNTk5XdUisU/FBxMRD407OVpLcY3Boh5NADhaTUHasgCO7T+8XW7dQ\nMik6OnH3xdWzGwlNuVmyMDVOVDBAnw6N6NOhEb+9sitrdx5ixupdzFyzk6c/XcvTn66lY2I8w1Ka\nMTS5KT1a1SdQWTcCRteChh2crSR5J+DQzkKJpMjX3Wuc5ykyxRwdX/ropG5L59yKJRRTDEsWpkYT\nEbo2r0vX5nW5b0gS2w4cY5Z7gvzl+Zv427yNJCbEnrwkt1/HRsRGRXhFwKhYaNDW2UqSnwuHdxU/\nOsneDt/Nd0Ywml/kteNKOYfiPq7dyEq410CWLIwppGX9Wozu147R/dpx8Gguc9c5J8g/XLaNfy/6\nnviYIIPcSrmDzmlCvVoRrpRbkmA01GvlbCUJ5cPh3W4SySwyStkO33/tnGMJFVm3PBjj3Lle7OjE\nTSjxic45G1NtWLIwpgT1akdzTc+WXNOzJcdz8/l6Y9bJ1QA/XrmDqIDQp8MPlXJb1K8V6ZDDEwg6\nV1vVbQ70Kr5NKARH98LBzNNHJ9nbYdsSWPMR5J8o8tpRbkJpceqopPDXOk0hWHM/gvLyQ+Tkh8jJ\nc7f8Il/d7UR+iNyi+wu13XbiIhrIYVJ8jrfm/ksZE4a46CCDuzRhcJcmPHVNN5ZtPXDyfo7xU1cz\nfupqurWsy7DkZgxLaco5TROqzoqAZyIQcNbyqNOk5GVnVeHovuJHJ9nbYOe3Tqn3vGOn9pMA1Gl2\n2uikbv4BQhJ0yqkgTjsJOOdSJFDKPim2nYqQG1Jy8iEvJOSGICcfckMhcvIhJ1/JzYfckJDj7juR\nr+5zkJuvnAgpOXnOvuNunxN56jyXr5zIhxN5zuOcvBC5RROB+8F+ouC5vBChCrtrYQBdgpk8VFEv\nVwK7z8KYM5ThVsqdmb6TZVsPoAqtG9ZiWLJzgrz229cQFPXtvgFVJaQQUiU/pGjBY1U09MPjkDrP\n5Yd+eFzQJ6TO6+SrEnL7hAq9rqqSX3j/KW2K7xNS5w5753klmHOQuKM7qXVsJ3HHd1Hr2E5qH991\ncos/sYuY/KO+HKPKFEJQBCXgfBUBBHUTmPO9k9S0aKJDIBBACj8utE9EQIKICBIIIIEAJ3Zv4kQg\njvrjtpQrXrspz5gI2H3oOLPX7GbG6p0syMgiJz9EVECIjQrQrnH8KR/mp36wnvqBmx8qSAJFPnQV\nJwkUPK64P0+rhDocpbnsow7HCAYgJgDRQSEmCLEBIToIMUEhOgDRAYgJOs9HByDGfT46oEQFIDrg\ntgs6baMFogJKVEDc53G+CgQD7nPi7Cv4GhQlKE4f5zGIhkBDgDpfFfdr4X1awj6t4HYhstfNJ0di\naDxuY7mOud2UZ0wENEmIY1TvNozq3YbDJ/KYv34P4z5cRV6+0rxeHCJCUIRAgB8eCwRECAR+eCwi\nBAPu/pMbbptCfYruO+35ovtPbxcMOO93Wh8R97nTHwfc+KTgsZzaLiCcfM2g+15S9HGhn00CP3y/\n9tlLCBLivMe+iPQ/51lhq3sHd2Of38eShTE+qRMbxeXnNufNrzYD8OroCyIb0FkiRvLLbmQqnV0s\nbYwxpkyWLIwxxpTJkoUxxpgyWbIwxhhTJksWxhhjymRXQxljqpQJjZ4F4J0Ix2FOZSMLY4wxZbKR\nhTE+s7WkjZ8qa/lZX0cWIjJCRNaJSIaIPFJCmxtFJF1EVovIvwvtHy0iG9xttJ9xGmOMKZ1vIwsR\nCQKTgKFAJrBYRKaqanqhNknAo0B/Vd0vIk3c/Q2B8UAqTuWVJW7f/X7Fa4ypGmwkVjX5ObLoDWSo\n6iZVzQEmA1cXaXMnMKkgCahqweLCw4GZqrrPfW4mMMLHWI0xxpTCz2TREtha6PtMd19hnYHOIrJA\nRBaKyIgw+hpjjKkkfp7gLm7ll6L1lKOAJGAQ0Ar4QkS6eeyLiNwF3AXQpk2bM4nVGGNMKfwcWWQC\nrQt93wrYXkybD1U1V1W/A9bhJA8vfVHVl1U1VVVTExMTKzR4Y4wxP/AzWSwGkkSkvYjEADcDU4u0\n+QAYDCAijXGmpTYB04FhItJARBoAw9x9xhhjIsC3aShVzRORMTgf8kHgNVVdLSITgDRVncoPSSEd\nyAceVtUsABGZiJNwACao6j6/YjXGGFM6W1bVGGNqMK/Lqlq5D2OMMWWyZGGMMaZM1WYaSkT2AFvO\n4CUaA3srKJyKZHGFx+IKj8UVnuoYV1tVLfNy0mqTLM6UiKR5mberbBZXeCyu8Fhc4anJcdk0lDHG\nmDJZsjDGGFMmSxY/eDnSAZTA4gqPxRUeiys8NTYuO2dhjDGmTDayMMYYU6YamyxE5AZ3db6QiJR4\nFYGX1f4qOK6GIjLTXSFwplsbq7h2+SKy3N2K1tyqyHhK/flFJFZE3nGfXyQi7fyKJYyYbhORPYWO\nz8/8jsl939dEZLeIrCrheRGR5924vxWR86tIXINE5GCh4zWukuJqLSJzRWSN+7t4XzFtKv2YeYyr\n0o+ZiMSJyDcissKN64li2vj3+6iqNXIDugLnAPOA1BLaBIGNQAcgBlgBJPsc1x+AR9zHjwDPlNDu\ncCUcozJ/fuAXwIvu45uBd6pATLcBL0Tg/9RA4HxgVQnPXw58ilOCvw+wqIrENQiYFoHj1Rw4332c\nAKwv5t+y0o+Zx7gq/Zi5x6CO+zgaWAT0KdLGt9/HGjuyUNU1qrqujGZeVvuraFcDb7qP3wSu8fn9\nSuPl5y8c7xTgUhEpbj2SyowpIlR1PlBawcurgbfUsRCoLyLNq0BcEaGqO1R1qfv4ELCG0xc5q/Rj\n5jGuSuceg8Put9HuVvSks2+/jzU2WXgUiRX7mqrqDnD+0wJNSmgXJyJp7gqDfiUULz//yTaqmgcc\nBBr5FI/XmACuc6ctpohI62Kej4SqvAJkX3d641MRSansN3enS3ri/LVcWESPWSlxQQSOmYgERWQ5\nsBtn6ekSj1dF/z76uVJexInILKBZMU89pqofenmJYvad8eVjpcUVxsu0UdXtItIBmCMiK1V145nG\nVoSXn9+XY1QKL+/3EfC2qp4QkXtw/tK6xMeYvKrsY+XVUpySD4dF5HKcdWaSKuvNRaQO8B/gflXN\nLvp0MV0q5ZiVEVdEjpmq5gM9RKQ+8L6IdFPVwueifDte1TpZqOqQM3wJTyv2hau0uERkl4g0V9Ud\n7nB7dwmvsd39uklE5uH89VPRycLraoetgUwRiQLq4e+UR5kxqbsmiusV4Bkf4wmHL/+fzlThD0JV\n/URE/ioijVXV9xpIIhKN84H8L1V9r5gmETlmZcUVyWPmvucB9/d+BFA4Wfj2+2jTUKXzstpfRZsK\njHYfjwZOGwGJs4JgrPu4MdAfSPchFi8/f+F4rwfmqHt2zSdlxlRkTnskzpxzVTAVuNW9wqcPcLBg\nyjGSRKRZwby2iPTG+VzIKr1XhbyvAH8H1qjq/5bQrNKPmZe4InHMRCTRHVEgIrWAIcDaIs38+32s\nzLP5VWkDrsXJwieAXcB0d38L4JNC7S7HuRpiI870ld9xNQJmAxvcrw3d/anAq+7jfsBKnCuBVgJ3\n+BjPaT8/MAEY6T6OA/4PyAC+ATpUwjEqK6bfA6vd4zMX6FJJ/6feBnYAue7/rTuAe4B73OcFmOTG\nvZISrsKLQFxjCh2vhUC/SorrIpwpkm+B5e52eaSPmce4Kv2YAecBy9y4VgHj3P2V8vtod3AbY4wp\nk01DGWOMKZMlC2OMMWWyZGGMMaZMliyMMcaUyZKFMcaYMlmyMCYMInK47Fal9p/i3nWPiNQRkZdE\nZKNbRXS+iFwoIjHu42p906w5u1iyMKaSuPWDgqq6yd31Ks7dtUmqmoJTLbexOgUSZwM3RSRQY4ph\nycKYcnDvKH5WRFaJyEoRucndH3BLP6wWkWki8omIXO92+zHuHfki0hG4EBirqiFwSreo6sdu2w/c\n9sZUCTbMNaZ8fgT0ALoDjYHFIjIfp/RKO+BcnIrBa4DX3D79ce6mBkgBlqtTGK44q4ALfIncmHKw\nkYUx5XMRTmXbfFXdBXyO8+F+EfB/qhpS1Z045UYKNAf2eHlxN4nkiEhCBcdtTLlYsjCmfEpaUKa0\nhWaO4dTuAaeuUHcRKe13MBY4Xo7YjKlwliyMKZ/5wE3uYjSJOEuXfgN8ibPwUkBEmuIsv1lgDdAJ\nQJ21R9KAJwpVL00Skavdx42APaqaW1k/kDGlsWRhTPm8j1P9cwUwB/i1O+30H5zKrquAl3BWWDvo\n9vmYU5PHz3AWwcoQkZU4a28UrNUwGPjE3x/BGO+s6qwxFUxE6qizglojnNFGf1Xd6a5BMNf9vqQT\n2wWv8R7wqJa9TrwxlcKuhjKm4k1zF6mJASa6Iw5U9ZiIjMdZJ/n7kjq7izp9YInCVCU2sjDGGFMm\nO2dhjDGmTJYsjDHGlMmShTHGmDJZsjDGGFMmSxbGGGPKZMnCGGNMmf4/7xnGtYcTaL8AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c3cf310>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    #plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "plt.legend()\n",
    "plt.xlabel( 'log(C)' )                                                                                                      \n",
    "plt.ylabel( 'logloss' )\n",
    "plt.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存模型，用于后续测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cPickle\n",
    "\n",
    "cPickle.dump(grid.best_estimator_, open(\"Otto_L1_org_tfidf.pkl\", 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
